Image synthesis method and image synthesis device for virtual object

ABSTRACT

An image synthesis method of a virtual object and the apparatus thereof are provided. The image synthesis method of the virtual object comprises providing a first depth image of a scene and a first two-dimensional image of the scene; providing a second depth image of the virtual object; adjusting a second depth value of the virtual object in the first depth image according to an objective location in the first depth image and a reference point of the second depth image; rendering a second two-dimensional image of the virtual object; and synthesizing the first two-dimensional image and the second two-dimensional image according to a lighting direction of the first two-dimensional image, an adjusted second depth value and the objective location in the first depth image.

This application claims the benefit of Taiwan application Serial No.105136870, filed Nov. 11, 2016, the disclosure of which is incorporatedby reference herein in its entirety.

TECHNICAL FIELD

The disclosure relates to an image synthesis method and an imagesynthesis device for a virtual object.

BACKGROUND

If one virtual object having depth information is needed to besynthesized with an image, the virtual object may be synthesized with adepth image which has depth information and sense of space or may besynthesized with a two-dimensional image. The two-dimensional image isobtained by capturing a real scene by a camera, so the two-dimensionalimage has good quality, and the edge, the color, the light pattern andshadow pattern of the two-dimensional image are approximate that of thereal scene.

However, because the two-dimensional image does not have any depthinformation, the virtual object synthesized with the two-dimensionalimage may have wrong sense of distance or cover an object which shouldbe located at the rear of the virtual object.

Therefore, to make sure the virtual object synthesized with thetwo-dimensional image have correct light pattern, correct shadowpattern, correct sense of distance and correct positional relationshipwithout affecting the image quality is an important issue nowadays.

SUMMARY

According to one embodiment, an image synthesis method for a virtualobject is provided. The image synthesis method includes the followingsteps: A first depth image of a scene and at least one firsttwo-dimensional image of the scene are provided. A second depth image ofthe virtual object is provided. A second depth value of the virtualobject is adjusted according to an object location of the first depthimage and a reference point of the second depth image. At least onesecond two-dimensional image of the virtual object is rendered accordingto an image lighting direction of the first two-dimensional image, thesecond depth value which is adjusted and the objection location of thefirst depth image. The first two-dimensional image and the secondtwo-dimensional image are synthesized.

According to another embodiment, an image synthesis device for a virtualobject is provided. The image synthesis device includes a camera, anobject providing unit and a processor. The camera is for providing afirst depth image of a scene and at least one first two-dimensionalimage of the scene. The object providing unit is for providing a seconddepth image of the virtual object. The processor includes a depthadjusting unit, a light source providing unit and a rendering unit. Thedepth adjusting unit is for adjusting a second depth value of thevirtual object according to an object location of the first depth imageand a reference point of the second depth image. The light sourceproviding unit is for analyzing an image lighting direction of the firsttwo-dimensional image. The rendering unit is for rendering at least onesecond two-dimensional image of the virtual object according to theimage lighting direction of the first two-dimensional image, the seconddepth value which is adjusted and the objection location of the firstdepth image and for synthesizing the first two-dimensional image and thesecond two-dimensional image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an image synthesis device for a virtual object according toan embodiment of the present disclosure.

FIG. 2 shows the image synthesis device for the virtual object accordingto another embodiment of the present disclosure.

FIG. 3 shows a flowchart of an image synthesis method for the virtualobject according to an embodiment of the present disclosure.

FIG. 4 illustrates a first two-dimensional image, a secondtwo-dimensional image, a first depth image, a second depth image, asynthesized depth image, an object location, a reference point and asynthesized two-dimensional image according to the image synthesismethod of FIG. 3.

FIG. 5 shows a flowchart of an analyzing method of the image lightingdirection of the first two-dimensional image according to an embodimentof the present disclosure.

FIG. 6 shows a flowchart of the step of classifying the pixels of thefirst two-dimensional image to be the groups.

FIG. 7 shows a classifying module according to one embodiment of thepresent disclosure.

In the following detailed description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the disclosed embodiments. It will be apparent,however, that one or more embodiments may be practiced without thesespecific details. In other instances, well-known structures and devicesare schematically shown in order to simplify the drawing.

DETAILED DESCRIPTION

FIG. 1 shows an image synthesis device for a virtual object according toan embodiment of the present disclosure. FIG. 2 shows the imagesynthesis device for the virtual object according to another embodimentof the present disclosure. The difference between FIG. 1 and FIG. 2 isin that a light source providing unit 132 of the FIG. 2 includes aclassifying module 132A and a calculating module 132B. FIG. 3 shows aflowchart of an image synthesis method for the virtual object accordingto an embodiment of the present disclosure. Please refer to FIGS. 2 and3, the image synthesis device 100 includes a camera 110, an objectproviding unit 120, and a processor 130. The camera 110 is used forproviding a first depth image D1 of a scene and at least one firsttwo-dimensional image I1 of the scene. For example, the camera 110 maybe a monochrome camera, or a combination of a color camera and aninfrared camera for providing the first two-dimensional image I1 whichis a monochrome image, a gray-scale image or a color image and the firstdepth image D1. The object providing unit 120 is used for providing asecond depth image D2 of the virtual object. For example, the objectproviding unit 120 may be an optical disc or a USB disk. A depth planecan be obtained from a third two-dimensional image of the virtual objectand the second depth image D2 is obtained according to the depth planeand the third two-dimensional image.

The processor 130 includes a depth adjusting unit 131, the light sourceproviding unit 132 and a rendering unit 133. The depth adjusting unit131 is used for adjusting a second depth value of the virtual objectaccording to an object location OL of the first depth image D1 and areference point RP of the second depth image D2. The light sourceproviding unit 132 is used for analyzing an image lighting direction ofthe first two-dimensional image I1. The rendering unit 133 is used forrendering at least one second two-dimensional image I2 of the virtualobject according to the second depth value which is adjusted, a firstdepth value of the objection location OL of the first depth image D1 andthe image lighting direction, and for synthesizing the firsttwo-dimensional image I1 and the second two-dimensional image I2. Theprocessor 130 is used for performing various procedures, such as depthadjusting procedure, light source providing procedure, and renderingprocedure. For example, the processor 130 may be a chip, a circuit, acircuit board, or a storage device storing a plurality of program codes.The object location OL of the first depth image D1 and the referencepoint RP of the virtual object at the second depth image D2 may be setby the user or may be predetermined in advance.

FIG. 4 illustrates the first two-dimensional image I1, the secondtwo-dimensional image I2, the first depth image D1, the second depthimage D2, a synthesized depth image D12, the object location OL, thereference point RP, and a synthesized two-dimensional image I12according to the image synthesis method of FIG. 3. The firsttwo-dimensional image I1, the second two-dimensional image I2, the firstdepth image D1, the second depth image D2, the synthesized depth imageD12, the object location OL, the reference point RP, and the synthesizedtwo-dimensional image I12 are shown. Please refer to FIGS. 2 to 4, inthe step S110, the first depth image D1 of the scene and the firsttwo-dimensional image I1 of the scene are provided. In one embodiment,the coordinate system of the first depth image D1 of the scene and thecoordinate system of the first two-dimensional image I1 of the scenecorrespond with each other. In one embodiment, the first depth image D1of the scene and the first two-dimensional image I1 of the scene may beprovided by the camera 110. According to the relationship between thefirst depth image D1 and the first two-dimensional image I1, the secondtwo-dimensional image I2 whose view angle is identical to that of thefirst two-dimensional image I1 can be obtained, when the virtual objectis rendered. The number of the at least one first two-dimensional imageI1 may be plurality and the number of the at least one secondtwo-dimensional image I2 may be plurality, and in the step rendering thesecond two-dimensional images I2 of the virtual object, the secondtwo-dimensional images I2 having different view angles are renderedaccording to a camera parameter corresponding the first two-dimensionalimages I1 having different view angles.

In the step S120, the second depth image D2 of the virtual object isprovided. In the step S130, a second depth value of the virtual objectat the first depth image D1 is adjusted according to an object locationOL of the first depth image D1 and a reference point RP of the seconddepth image D2. In one embodiment, please referring FIG. 3, the depthvalue of the first depth image D1 ranges from 0 to 255, the depth valueof the object location OL of the scene (the first depth image D1) is100; the depth value of the second depth image D2 ranges from 0 to 20,the reference point RP of the virtual object (of the second depth imageD2) is 10. When the depth adjusting unit 131 synthesizes the seconddepth image D2 and the first depth image D1 (for example, the virtualobject and the scene are synthesized to be a synthesized depth imageD12), the reference point RP of the virtual object (in the second depthimage D2) is aligned to the object location OL of the scene (in thefirst depth image D1). The depth value of the reference point RP of thevirtual object (in the second depth image D2) is changed from 10 to 110.As such, the depth value of the virtual object of the scene (in thefirst depth image D1) is changed and ranges from 100 to 120. Afterperforming this step, the location of the virtual object is confirmed.For example, in the synthesized depth image D12, part of the virtualobject is covered by the computer screen.

In the step S120 of providing the second depth image D2 of the virtualobject, a plurality of fourth two-dimensional images having differentview angles are further provided. In the step of rending the secondtwo-dimensional image I2, one of the fourth two-dimensional imageshaving a particular view angle is selected to be rendered. If the fourthtwo-dimensional image having the particular view angle is not existed,then the fourth two-dimensional image has the particular view angle maybe obtained by interpolation.

In the step S140, the second two-dimensional image I2 of the virtualobject is rendered according to the image lighting direction of thefirst two-dimensional image I1, the second depth value which is adjustedand the first depth value at the objection location. The depth value ofthe virtual object is updated to be 110 in the forgoing embodiment. Thedepth value of the object location OL of the scene (in the first depthimage D1) is updated to be 100 in the forgoing embodiment. The lightingdirection of the first two-dimensional image I1 is illustrated in FIG.5.

In the step S150, the first two-dimensional image I1 and the secondtwo-dimensional image I2 are synthesized to be the synthesizedtwo-dimensional image I12. If part of the virtual object is hidden byone object of the first two-dimensional image I1, such as the computerscreen of the synthesized depth image D12 in the FIG. 4, the portion ofthe virtual object which is hidden, such as the lower left part of thesecond two-dimensional image I2 of the FIG. 4, is hollowed out, when thesecond two-dimensional image I2 of the virtual object is rendered.

FIG. 5 shows a flowchart of an analyzing method of the image lightingdirection of the first two-dimensional image I1 according to anembodiment of the present disclosure. Please refer to FIGS. 2 to 5. Thelight source providing unit 132 of the processor 130 is used foranalyzing the image lighting direction of the first two-dimensionalimage I1. The light source providing unit 132 includes the classifyingmodule 132A and the calculating module 132B. The image lightingdirection of the first two-dimensional image I1 can be obtained by aLambertian reflectance model, illustrated as the following equation (1):

I=ρλ(NL)  (1)

During analyzing the image lighting direction of the firsttwo-dimensional image I1 according to the equation (1), one pixel is aunit of measurement. I is the intensity of the actual color of thepixel, which may be provided by the camera 110. N is the surface normalvector of the pixel, ρ is the albedo of the surface, λ is the intensityof the illuminant, and ρλ is the average color of the actual color ofone group. L is a pixel lighting direction. During calculating the imagelighting direction of the first two-dimensional image I1, the locationof the light source is fixed, and the equation (1) can be differentiatedto I to obtain the following equation (2):

dI=d(ρλ(N·L))  (2)

Thus, the changes of the color can be calculated based on the surfacenormal vector. In the equation (2), only L in unknown, so the pixellighting direction can be obtained.

Further, during calculating the image lighting direction of the firsttwo-dimensional image I1, the classifying module 132A classifies thepixels of the first two-dimensional image I1 to be a plurality of groupsaccording to the actual colors of the pixels of the firsttwo-dimensional image I1. The calculating module 132B obtains thesurface normal vector (N) of each of the pixels in the firsttwo-dimensional image I1 according to the first depth image D1 of thescene. In one embodiment, during calculating the surface normal vector(N), the first depth image D1 can be transformed to be a 3D triangularmesh, and then the surface normal vector (N) of each pixel can becalculated according to the 3D triangular mesh. The calculating module132B calculates the average color (ρΔ) of each of the groups. Thecalculating module 132B is further analyzes the pixel lighting direction(L) of each of the pixels according to the surface normal vector (N) ofeach of the pixels, the actual color (i) of each of the pixels and theaverage color (ρλ) of each of the groups, and obtains the image lightingdirection of the first two-dimensional image I1 by averaging the pixellighting directions of the pixels.

FIG. 6 shows a flowchart of the step of classifying the pixels of thefirst two-dimensional image to be the groups. FIG. 7 shows theclassifying module according to one embodiment of the presentdisclosure. Please refer to FIGS. 1 to 7. The classifying module 132Aincludes a selecting element G1, a first determining element G2, aclassifying element G3, a second determining element G4, an excludingelement G5 and a third determining element G6. The classifying module132A is used for performing the following steps. In one embodiment, theselecting element G1 selects an unclassified pixel from the pixels ofthe first two-dimensional image I1. A target group of the groupscontains the unclassified pixel. The first determining element G2searches a surrounding pixel which surrounds with the target group anddetermines whether the actual color of the surrounding pixel is similarto the actual color of the target group. For example, the firstdetermining element G2 searches 8 surrounding pixels which surround theunclassified pixel. The unclassified pixel and the 8 surrounding pixelsform a 3*3 square. If the actual color of one of the surrounding pixelsis similar to the actual color of the target group, the classifyingelement G3 classifies this surrounding pixel to be the target group. Thesecond determining element G4 determines whether there is anothersurrounding pixel which surrounds with the target group. If there is nosurrounding pixel, the excluding element G5 excludes an extreme pixelhaving an extreme color from the target group; if there is anothersurrounding pixel, the first determining element G2 searches anothersurrounding pixel which surrounds with the target group again anddetermines whether the actual color of the surrounding pixel is similarto the actual color of the target group. For example, if the targetgroup is a 2*3 matrix, then the first determining element G2 search 14surrounding pixels which surround the target group. The target group andthe surrounding pixels form a 4*5 matrix. If the first determiningelement G2 concludes that the actual color of the surrounding pixel isnot similar to the actual color of the target group, then the seconddetermining element G4 determines whether there is another surroundingpixel. The third determining element G6 determines whether all of thepixels of the first two-dimensional image I1 are classified. If all ofthe pixels of the first two-dimensional image I1 are classified, thenthe step of classifying one the pixels to be one of the groups isterminated; if one of the pixels of the first two-dimensional image isnot classified, then the selecting element G1 selects anotherunclassified pixel of the first two-dimensional image I1 again.

Further, the step of excluding the extreme pixel is illustrated asfollowing. For each of the groups, a statistic analysis of the actualcolor is performed. In one embodiment, the actual color of each pixel ofthe first two-dimensional image I1 of the scene includes three primarycolors, such as red color, blue color and green color. The intensitydistribution of the red color, the blue color, the green color of thepixels is drawn and the standard deviation of each of the three primarycolors is calculated. Whether the red color of one of the pixels islarger than a predetermined red value, the blue color of one of thepixels is larger than a predetermined blue color, or the green color ofone of the pixel is larger than a predetermined green value isdetermined. For example, in the intensity distribution of the red color,a red color standard deviation is obtained. In one embodiment, if onepixel has a color which is larger than 2 times the standard deviation,then this pixel has an extreme color and this pixel can be excluded fromthe group. The three primary colors of one pixel may have differentvalues. In one embodiment, if any one of the red color, the blue colorand the green color of one pixel are larger than 2 times the blue colorstandard deviation and the green color standard deviation respectively,then this pixel is excluded from the group. In one embodiment, thepredetermined red value is N times a red color standard deviation in oneof the groups, the predetermined blue value is N times a blue colorstandard deviation in one of the groups, the predetermined green valueis N times a green color standard deviation in one of the groups, and Nis rational number and larger than zero. In another embodiment, one ofthe pixel having more than two colors which are larger than 2 times thestandard deviations respectively is excluded. The excluded pixel isneeded to be classified again.

In practical applications, the image synthesis method for the virtualobject of the present disclosure can be used in medical operation,Augmented Reality (AR) games, display of new building or furnituredisplay in interior design.

In the image synthesis method and the image synthesis device for thevirtual object of the present disclosure, the virtual object issynthesized with the first two-dimensional image. The virtual object isrendered according to the light distribution of the firsttwo-dimensional image, the shading and the space sense, such that thedimensional sense of the virtual object synthesized with the scene canbe improved.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed embodiments.It is intended that the specification and examples be considered asexemplary only, with a true scope of the disclosure being indicated bythe following claims and their equivalents.

What is claimed is:
 1. An image synthesis method for a virtual object,comprising: providing a first depth image of a scene and at least onefirst two-dimensional image of the scene; providing a second depth imageof the virtual object; adjusting a second depth value of the virtualobject according to an object location of the first depth image and areference point of the second depth image; rendering at least one secondtwo-dimensional image of the virtual object according to an imagelighting direction of the first two-dimensional image, the second depthvalue which is adjusted and a first depth value at the objectionlocation of the first depth image; and synthesizing the firsttwo-dimensional image and the second two-dimensional image.
 2. The imagesynthesis method according to claim 1, wherein the step of rendering thesecond two-dimensional image of the virtual object comprises: hollowingout a portion of the virtual object which is hidden from one object ofthe first two-dimensional image, when the second two-dimensional imageof the virtual object is rendered.
 3. The image synthesis methodaccording to claim 1, wherein a number of the at least one firsttwo-dimensional image of the scene is plurality, and the firsttwo-dimensional images have different view angles.
 4. The imagesynthesis method according to claim 3, wherein in the step of renderingthe at least one second two-dimensional image of the virtual object, anumber of the at least one second two-dimensional image is plurality,and the second two-dimensional images having different view angles arerendered according to a camera parameter.
 5. The image synthesis methodaccording to claim 1, wherein in the step of providing the second depthimage of the virtual object, a plurality of the fourth two-dimensionalimages having different view angles are provided, and in the step ofrendering the at least one second two-dimensional image of the virtualobject, one of the fourth two-dimensional images having a particularview angle is selected to be rendered.
 6. The image synthesis methodaccording to claim 5, wherein the one of the fourth two-dimensionalimages has the particular view angle is obtained by interpolation. 7.The image synthesis method according to claim 1, wherein the imagesynthesis method further comprises obtaining the image lightingdirection of the first two-dimensional image and the step of obtainingthe image lighting direction of the first two-dimensional imagecomprises: obtaining a surface normal vector of each of a plurality ofpixels in the first two-dimensional image according to the first depthimage of the scene; obtaining an actual color of each of the pixelsaccording to the first two-dimensional image of the scene; classifyingthe pixels to be a plurality of groups according to the actual colors ofthe pixels; calculating an average color of each of the groups;analyzing a pixel lighting direction of each of the pixels according tothe surface normal vectors of the pixels, the actual colors of thepixels and the average colors of the groups; and obtaining the imagelighting direction of the first two-dimensional image by averaging thepixel lighting directions of the pixels.
 8. The image synthesis methodaccording to claim 7, wherein the step of classifying one of the pixelsto be one of the groups comprises: selecting an unclassified pixel fromthe pixels of the first two-dimensional image, wherein a target group ofthe groups contains the unclassified pixel; searching a surroundingpixel which surrounds with the target group and determining whether theactual color of the surrounding pixel is similar to the actual color ofthe target group; classifying the surrounding pixel to be the targetgroup, if the actual color of the surrounding pixel is similar to theactual color of the target group; determining whether there is anothersurrounding pixel which surrounds with the target group; excluding anextreme pixel having an extreme color from the target group, if there isno surrounding pixel; and determining whether all of the pixels of thefirst two-dimensional image are classified, wherein if all of the pixelsof the first two-dimensional image are classified, then the step ofclassifying one of the pixels to be one of the groups is terminated. 9.The image synthesis method according to claim 8, wherein after the stepof determining whether the actual color of the surrounding pixel issimilar to the actual color of the target group is performed, if theactual color of the surrounding pixel is not similar to the actual colorof the target group, then the step of determining whether there isanother surrounding pixel is performed.
 10. The image synthesis methodaccording to claim 9, wherein after the step of determining whetherthere is another surrounding pixel is performed, if there is anothersurrounding pixel, then the step of searching the surrounding pixel anddetermining whether the actual color of the surrounding pixel is similarto the actual color of the target group is performed.
 11. The imagesynthesis method according to claim 9, wherein after the step ofdetermining whether all of the pixels of the first two-dimensional imageare classified is performed, if at least one of the pixels of the firsttwo-dimensional image is not classified, then the step of selecting theunclassified pixel is performed.
 12. The image synthesis methodaccording to claim 9, wherein in the step of excluding the extremepixel, a red color of the extreme pixel is larger than a predeterminedred value, a blue color of the extreme pixel is larger than apredetermined blue color, or a green color of the extreme pixel islarger than a predetermined green value.
 13. The image synthesis methodaccording to claim 12, wherein the predetermined red value is N times agreen color standard deviation in one of the groups, the predeterminedblue value is N times a blue color standard deviation in one of thegroups, the predetermined green value is N times a green color standarddeviation in one of the groups, and N is rational number and larger thanzero.
 14. The image synthesis method according to claim 1, wherein acoordinate system of the first depth image of the scene and a coordinatesystem of the first two-dimensional image of the scene correspond witheach other.
 15. An image synthesis device for a virtual object,comprising: a camera for providing a first depth image of a scene and atleast one first two-dimensional image of the scene; an object providingunit for providing a second depth image of the virtual object; and aprocessor, including: a depth adjusting unit for adjusting a seconddepth value of the virtual object according to an object location of thefirst depth image and a reference point of the second depth image; alight source providing unit for analyzing an image lighting direction ofthe first two-dimensional image; and a rendering unit for rendering atleast one second two-dimensional image of the virtual object accordingto the image lighting direction of the first two-dimensional image, thesecond depth value which is adjusted and a first depth value at theobjection location of the first depth image and for synthesizing thefirst two-dimensional image and the second two-dimensional image. 16.The image synthesis device according to claim 15, wherein when therendering unit renders the second two-dimensional image of the virtualobject, a portion of the virtual object which is hidden is hollowed outfrom one object of the first two-dimensional image.
 17. The imagesynthesis device according to claim 15, wherein the light sourceproviding unit comprising: a classifying module for classifying aplurality of pixels of the first tow-dimensional image to be a pluralityof groups according to a plurality of actual colors of the pixels; and acalculating module for obtaining a surface normal vector of each of thepixels in the first two-dimensional image according to the first depthimage of the scene, for calculating an average color of each of thegroups, for analyzing a pixel lighting direction of each of the pixels,and for obtaining the image lighting direction of the firsttwo-dimensional image by averaging the pixel lighting directions of thepixels.
 18. The image synthesis device according to claim 17, whereinthe classifying module includes: a selecting element for selecting anunclassified pixel from the pixels of the first two-dimensional image,wherein a target group of the groups contains the unclassified pixel; afirst determining element for searching a surrounding pixel whichsurrounds with the target group and for determining whether the actualcolor of the surrounding pixel is similar to the actual color of thetarget group; a classifying element for classifying the surroundingpixel to be the target group, if the actual color of the surroundingpixel is similar to the actual color of the target group; a seconddetermining element for determining whether there is another surroundingpixel which surrounds with the target group; an excluding element forexcluding an extreme pixel having an extreme color from the targetgroup, if there is no surrounding pixel; and a third determining elementfor determining whether all of the pixels of the first two-dimensionalimage are classified, wherein if all of the pixels of the firsttwo-dimensional image are classified, then the classifying module stopsclassifying.
 19. The image synthesis device according to claim 18,wherein if the first determining element concludes that the actual colorof the surrounding pixel is not similar to the actual color of thetarget group, then the second determining element starts to determinewhether there is another surrounding pixel.
 20. The image synthesisdevice according to claim 18, wherein if the second determining elementconcludes that there is another surrounding pixel, then the firstdetermining element searches the surrounding pixel again and determineswhether the actual color of the surrounding pixel is similar to theactual color of the target group again.
 21. The image synthesis deviceaccording to claim 18, wherein if the third determining elementconcludes that at least one of the pixels of the first two-dimensionalimage is not classified, then the selecting element selects theunclassified pixel again.
 22. The image synthesis device according toclaim 18, wherein a red color of the extreme pixel is larger than apredetermined red value, a blue color of the extreme pixel is largerthan a predetermined blue color, or a green color of the extreme pixelis larger than a predetermined green value.
 23. The image synthesisdevice according to claim 22, wherein the predetermined red value is Ntimes a red color standard deviation in one of the groups, thepredetermined blue value is N times a blue color standard deviation inone of the groups, the predetermined green value is N times a greencolor standard deviation in one of the groups, and N is rational numberand larger than zero.